Queueing Messages using Celery with RabbitMQ Message Broker Server

RabbitMQ & Celery Tutorials

We have already installed RabbitMQ and Celery in previous chapter: Installing RabbitMQ & Celery.

The picture below demonstrates how RabbitMQ works:

Picture from slides.com.

When we have a Celery working with RabbitMQ, the diagram below shows the work flow.

Picture from AMQP, RabbitMQ and Celery - A Visual Guide For Dummies.

Though Celery provides us lots of features, in this tutorial, we're going to deal with only the minimal basics.

We need to create a Celery instance in order to use celery's task queuing capabilities. This is a simple process of importing the package, creating an "app", and then setting up the tasks that celery will be able to execute in the background.

Here is our code, task.py:

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task(ignore_result=True)
def print_hello():
    print 'Hello World!'

@app.task
def gen_prime(x):
    multiples = []
    results = []
    for i in xrange(2, x+1):
        if i not in multiples:
            results.append(i)
            for j in xrange(i*i, x+1, i):
                multiples.append(j)
    return results

We may want to put our worker process ready for work in the background:

$ celery worker -A tasks &
...
 -------------- celery@k v3.1.13 (Cipater)
---- **** ----- 
--- * ***  * -- Linux-3.13.0-27-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x7f3f967d4610
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     amqp
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery
...

Note that the exchange type is direct and the key is celery.

We spawned the worker process in background. We can see how it works using Python interpreter:

>>> from tasks import print_hello
>>> from tasks import gen_prime
>>> print_hello()
Hello World!
>>> primes = gen_prime(1000)
>>> print primes
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683, 691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769, 773, 787, 797, 809, 811, 821, 823, 827, 829, 839, 853, 857, 859, 863, 877, 881, 883, 887, 907, 911, 919, 929, 937, 941, 947, 953, 967, 971, 977, 983, 991, 997]

Not sure it's using background worker or not. However, if we run time consuming task by putting a big number, it will hang if not running on background worker:

>>> primes = gen_prime(100000)

Indeed! Clearly, it's not using the worker in the background. But there is a way to use it: .delay() method:

>>> primes = gen_prime.delay(100000)

It immediately give us the prompt back, and we can check the status of the function as well.

>>> primes.ready()
False
>>> primes.ready()
False
>>> primes.ready()
True
>>> print primes.get()
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67
....

1. How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS
2. AMQP, RabbitMQ and Celery - A Visual Guide For Dummies

RabbitMQ & Celery Tutorials